Text Article Detection - #91
Conversation
Huge refactor: entire codebase in PEP8, imports alphabetized, bugfixes, core changes
`Setup.py` used to require a package directory named `data`, this directory was removed in a previous commit.
Reference this fix in the repo of a library we used to be dependent on: grangier/python-goose#114
…ill now run for everyone!) Deleted a bunch of unused stray data files under /tests/data.
…_html=True was used. On adding the import the problem was resolved
Added link to basic demo
Parse was breaking in the method clean_article_html when keep_article_ht...
split title with _
…ewspaper into igor-shevchenko-slash-splitter Conflicts: newspaper/extractors.py
|
Hey, thanks for the PR. I see from your above comment that you've made changes that have helped our full-text extraction: However, I think you may have messed up your pull request content, a lot of old commits are present in this pull request (none are from your Github account, which makes me think that you have local changes and probably have forgotten to push them?) Regardless, the top of this PR reads: You are trying to merge a bunch of commits from the master branch of this repo into a stale branch from a while ago! |
|
Sorry guy I just wanted to make a comment not doin any change on what I observed from article detection for Goose and I really messed up I'm sorry! |
5 participants
Hello guy,
Thanks for this beautiful work. I know that the code is implemented for "newspaper" and adapted to it.
I twisted a little bit the code juste to get the parser extract the main content of webpages.( I recognize that you clarify a lot the Goose extraction system) but I noticed a slight problem for article detection in blogs.
I give an example:
http://blogs.nottingham.ac.uk/
the text extracted there is just the comment section
http://blogs.nottingham.ac.uk/makingsciencepublic/category/responsible-innovation/
th text extracted is only the about section: means for the entire blog categories it will detect the duplicate version of the about section
Just a question of curiosity, do you have any idea of making the self.top_node detection be wider and wiser? I didn't dare to reimplement it... Any suggestion is welcomed